Learning Low-Precision Structured Subnetworks Using Joint Layerwise Channel Pruning and Uniform Quantization
نویسندگان
چکیده
Pruning and quantization are core techniques used to reduce the inference costs of deep neural networks. Among state-of-the-art pruning techniques, magnitude-based algorithms have demonstrated consistent success in reduction both weight feature map complexity. However, we find that existing measures neuron (or channel) importance estimation for such procedures at least one two limitations: (1) failure consider interdependence between successive layers; and/or (2) performing a parametric setting or by using distributional assumptions on maps. In this work, demonstrate rankings output neurons given layer strongly depend sparsity level preceding layer, therefore, naïvely estimating drive will lead suboptimal performance. Informed observation, propose purely data-driven nonparametric, channel strategy works greedy manner based activations previous sparsified layer. We our proposed method effectively combination with statistics-based generate low precision structured subnetworks can be efficiently accelerated hardware platforms as GPUs FPGAs. Using algorithms, increased performance per memory footprint over solutions across range discriminative generative
منابع مشابه
Learning vector quantization for heterogeneous structured data
In this paper we introduce an approach to integrate heterogeneous structured data into a learning vector quantization. The total distance between two heterogeneous structured samples is defined as a weighted sum of the distances in the single structural components. The weights are adapted in every iteration of learning using gradient descend on the cost function inspired by Generalized Learning...
متن کاملLearning Sparse Structured Ensembles with SG-MCMC and Network Pruning
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing. In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks. In the first stage, we run SG-MCMC with group sparse priors to draw an ensemble of samples from the posterior dist...
متن کاملthe relationship between using language learning strategies, learners’ optimism, educational status, duration of learning and demotivation
with the growth of more humanistic approaches towards teaching foreign languages, more emphasis has been put on learners’ feelings, emotions and individual differences. one of the issues in teaching and learning english as a foreign language is demotivation. the purpose of this study was to investigate the relationship between the components of language learning strategies, optimism, duration o...
15 صفحه اولEnsemble Pruning Using Reinforcement Learning
Multiple Classifier systems have been developed in order to improve classification accuracy using methodologies for effective classifier combination. Classical approaches use heuristics, statistical tests, or a meta-learning level in order to find out the optimal combination function. We study this problem from a Reinforcement Learning perspective. In our modeling, an agent tries to learn the b...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Applied sciences
سال: 2022
ISSN: ['2076-3417']
DOI: https://doi.org/10.3390/app12157829